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ABSTRACT 



The purpose of this publication is to highlight the 
importance of monitoring and documenting alcohol and drug abuse (AOD) 
program activities. Section 1 of this publication covers basic ideas 
in evaluating AOD program implementation, various methods of 
gathering data, and suggestions on selecting instruments. Section 2 
contains abstracts of several instruments which have been used by 
school districts and evaluators to monitor and evaluate program 
implementation. Eleven of the instruments reviewed and summarized are 
paper-and-pencil questionnaires, checklists, and interview guides. 
Three are microcomputer software programs. Two examples of 
record-keeping forms are also included. All the instruments reviewed 
have been developed by researchers or practitioners in the last 9 
years and have been used in a school setting. Section 3 of this 
publication lists additional resources such as titles of journals and 
manuals that furnish more detailed explanations of particular topics 
as well as names and telephone numbers of resource centers which 
educators and community representatives may contact to seek 
additional assistance. (ABL) 
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Preface 



This Consumer's Guide was developed to highlight the importance of evaluating the implement- 
ation of AOD programs and to share information on available instruments. The audience for this 
Guide is prevention specialists and school-community teams in the Western region who are 
interested in monitoring and documenting their program operations and activities. 

The questionnaires and data-collection forms reviewed in the following pages represent only a 
small sample of instmments in the field. All the instruments reviewed, however, can be obtained 
easily and have been used previously by educators in various parts of the country. It should be 
noted that the Western Regional Center for Drug-Free Schools and Communities does not 
endorse any specific instruments or products. We encourage interested readers to contact the 
authors directly to ask for more information about any of the instruments summarized. 

The development team for this consumer's guide consisted of Dr. Flora Yen of the Western 
Regional Center and Dr. Margaret Page Kalkowski of our Laboratory's Evaluation and 
Assessment Program. The layout and graphics were done by Marjorie Wolfe. 

Special thanks are extended to the authors of the instruments summarized, and their agencies, for 
their courtesy and cooperation on this project. We ;^je also grateful to Dr. Carol Thomas of the 
Southwest Regional Laboratory and Mr. Ralph Bak^*.r of the Far West Laboratory for their review 
of the draft and for their valuable input 

Finally, I wish to take this opportunity to invite practitioners and evaluators who may have 
developed instruments evaluating program operations to send them to me so that their abstracts 
may be included in future editions of this Guide. 
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Overview 



The effectiveness of school-based alcohol and other drug (AOD) prevention programs has 
increasingly come under scrutiny (Klitzner, 1987; United States General Accounting Office, 1987, 
1990, 1992). Most of the attention and call for evaluation, however, have focused on program 
outcomes. Rarely have the discussions centered on evaluating the implementation of AOD 
programs. An exception is the implementation st?:dy conducted by the Research Triangle Institute 
in 1991, which was funded by the U.S. Department of Education. In order, however, to detect 
any program effects and then to attribute these effects to the program, both program outcomes 
and program implementation have to be evaluated. 

In 1990, the Western Regional Center for Drug-Free Schools and Communities developed a 
publication summarizing an array of student AOD survey instruments currently being used in the 
United States. Now in its second edition, the publication titled Surveys ofStudert Alcohol and 
Other Drug Use: A Consumer's Guide provides guidance to local schools and communities on 
how to select a survey instrument to assess program outcomes, including the extent of AOD use 
among youth. Since evaluating program operations was not addressed in that publication, the 
Western Regional Center staff began to plan and develop this particular publication tided 
Evaluating AOD Program Implementation: A Consumer's Guide, The purpose of this 
publication is to highlight the importance of monitoring and documenting program activities. 

Section 1 of this publication covers basic ideas in evaluating AOD program implementation, 
various methods of gathering data, and suggestions on selecting instruments. 

Section 2 of the publication contains abstracts of several instruments which have been used by 
school districts and evaluators to monitor and evaluate program implementation. Some of the 
instruments reviewed and summarized are paper-and-pencil questionnaires, checklists, and 
interview guides. Others are microcomp'Uer software programs. Examples of record-keeping 
forms are also included. 

Section 3 of this publication lists additional resources such as tides of journals and manuals that 
furnish more detailed explanations of particular topics as well as names and telephone numbers of 
resource centers which educators and community representatives in the Western region can 
contact to seek additional assistance. 
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Basic Concepts 



What is it? 

Evaluating program implementation (sometimes referred to as "formative" or "process" 
evaluation) looks at these aspects of a program: 

• the extent to which a program reaches the target population; 

• monitoring of program "dose", in terms of frequency of delivery and/or participation 
in program activities; 

• monitoring the organizational context or situational variability within which the 
program is implemented; 

• the extent to which programs or services are implemented so as to meet program 
goals; and 

• cost of program implementation 

(McGraw, McKinlay, McClements, Lasater, Assaf and Carleton, 1989) 



Why is it important? 

Looking at program outcomes without examining program processes makes it very difficult to: 

1) understand why the program worked, or why it did not; 

2) identify program strengths as well as problem areas which can be corrected; and 

3) attribute any success or failure in outcomes to the program itself. 

To illustrate the importance of gathering information on implementation, let's look at the three 
scenarios on the next page. 
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Scenario 1: Car Going Through a Tunnel 




Scenario 2: Car Going Through a Car Wash 




Scenario 3: Car Getting Repaired 
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In the first scenario, a car is going through a tunnel. In the second scenario, a car is going 
through a car wash, and in the third scenario, a car is undergoing repairs at a body shop. Each 
scenario depicts a different level of intervention-from none (tunnel), to moderate (car wash), nnd 
then extensive (body shop). Obviously, one would not expect any change in the car that just 
passed through a tunnel, but one would expect some change in the car that went through a car 
wash, and even more change after mechanics have fixed it. So it is with AOD program 
implementation and student outcomes. Some programs result in very low level of intervention 
while others provide a high level of intervention. An example of a low level of intervention is 
when 5>tudents in a school receive only one or two exercises from a drug prevention curriculum of 
thirty lessons, due to madequate preparation of its teachers or insufficient classroom time to teach 
the materials. 

In contrast, another school in the same district may have ensured that the teachers were 
adequately trained and motivated to use the curriculum material so that the result was very 
different. All the students at this school received the full sequence of lessons. Without checking 
this out, a district administrator not personally involved in delivering the cuiriculum really would 
not know that the level of intervention was so different for the two schools. 

Should the year-end assessment of students' knowledge of AOD prevention from the two schools 
be combined, the findings surely would be difficult to understand and could often lead to errors in 
judging the effectiveness of the curriculum. 

What program components should be evaluated? 

There are basically two sets of questions that one can ask about program implementation. One 
set refers to the degree of implementation of a program. (Scheirer and Rezmovic, 1983). This 
would include how the key features of the program are being carried out. These key features can 
usually be identified from program designs. For example, if an AOD program consists of a 
prevention curriculum for all students, then the following questions can be asked: 

1) How many teachers used the curriculum? 

2) How many lessons from the curriculum were taught? 

3) What was the average amount of time each lesson took? 

4) When were these lessons taught? 

5) How many students at each grade level received ihe curriculum? 

Answers to these questions would indicate the extent of use of the curriculum at that particular 
school 

The second set of questions refers to the context or implementation processes (King, Morris, and 
Fitz-Gibbon, 1987; Scheirer and Rezmovic, 1983) in which a program may take place. For 
example, the context may include the organizational structure, the school, administrative and staff 
support for the curriculum, and attitudes toward AOD issues. In other words, these are 
contextual elements which may facilitate or hinder the use of the curriculum and its possible 
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effects on students. Some questions on context or implementation processes which may be asked 
are: 

1) What were teachers' attitudes toward the curriculum itself? 

2) Hov/ many teachers are sharing one set of curriculum materials? 

3) What is the extent of administrative support for the curriculum? 

4) What are students' attitudes toward AOD use? 

5) What are community attitudes toward AOD use? 

Getting answers to the second set of questions will help to illuminate how well or poorly the 
curriculum is being implemented and received at the site and provide a clearer picture of what 
program outcomes to realistically expect. 
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Gathering Information 



Although researchers and evaluators have acknowledged the importance of evaluating program 
implementation for over ten years now, its conceptualization and methodology are still not fully 
developed (Lipps and Grant, 1990; Pentz, Trebow, Hansen, MacKinnon, Dwyer, Johnson, Flay, 
Daniels, and Cormack, 1990; Scheirer and Rezmovic 1983). Gt t;raily, the methods used have 
been informal and utilize botJi quantitative and qualitative approaches to gathering information. 
*rhe three methods often used are: 

• self-report measures such as questionnaires or interviews 

• conducting observations 

• examining existing records 

Each of these methods have their advantages as well as disadvantages. These pros and cons are 
summarized below: 



Advantages and Disadvantages of Various Data 
Collection Methods 



Questionnaires: 



Advantages: 



Easy to administer. 

Easy to tabulate responses. 

Answers remain anonymous. 



Disadvantages: 



No opportunity to clarify questions or responses. 
Requires respondents to read and write. 
Easy to fake responses. 



Interviews: 



Advantages: 



Questions can be clarified when needed. 

Unclear responses can be pursued. 

Does not require respondents to read and write. 



Disadvantages: 



Time-consuming to administer. 
Time-consuming to summarize responses. 
Requires training of interviewers. 
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Observations: 

Advantages: Provides accounts of actual occurrence. 

Does not rely on respondent's recall. 
Does not rely on responses of respondents. 

Disadvantages: Intrusive. 

Requires training of observers. 
Errors in coding may occur. 



Existing Records: 

Advantages: Wide variety of information is readily available. 

Relatively low cost to obtain. 
Does not require much training to collect. 

Disadvantages: Records may be incomplete. 

Legal requirements may arise. 

May require time and effort to extract information needed. 



Suggestions and tips from the Western Regional Center's 1988 planning manual on using these 
methods are provided on pages 1.8 through 1.14. 
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Development of Questionnaire and Interview Instruments 



Planning 

Specific questions used in self-report instruments should be prepared in the context of a total 
evaluation plan. Many questionnaires and interviews yield useless information, simply because 
the evaluator did not take the time to construct a plan. The foUo^'ing steps are suggested in 
planning your self-report instruments. 

1. Detennine what information is to be sought. 

2. Determine what format the instrument will take. 

3. Develop the first draft. 

4. Revise the first draft. 

5. Pilot- test the instrument and revise if necessary. 

6. Write the detailed procedures for use. 

Question Development 

1. is the question necessary? 

2. Is more than one question necessary to get the information you need? 

3. Do respondents have enough knowledge about the question to be able to answer? 

4. Is tiie question biased? 

5. Will die content of die question offend, embarrass, or otherwise lead the respondent to 
not want to answer? 

. 6. Is the wording clear? 

1, Is the language level appropriate? 

8. Do the words evoke an emotional response? 
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Sample Interview Questions 



For District/School Site Program Administrators: 

1. Describe the implementation of the activities, management procedures, and 
budget allocations of the program. 

2. How have you departed from your intended plan, if at all, in these three areas? 
Why have you departed? 

3. What particular problems have you encountered? 
For Teachers: 

1. How were you informed of the program and your role in implementing it? 

2. Describe your role in the implementation of the program. 

3. Did you receive training and/or assistance in carrying out your role? If so, 
please describe. Was it adequate? 

4. Did the program operate smoothly and efficiently in the classroom? If not, 
why not? 

5. Was the program well-managed? If not, why not? 
For Students: 

1. Describe the program to me. Tell me about the topics that were covered and 
how the material was presented to you. 

2. Did your teacher seem comfortable and well prepared in presenting the 
material? 

3. Were there any incidents that occurred while the program was being carried 
out that stand out in your mind.? 

4. Did you enjoy the program? Was it interesting? Did you learn from it? 
For Parents: 

1. Are you aware of the program which is being carried out in the district/school? 
If so, how did you find out about it? 

2. Were you involved in the planning or carrying out of tlie program? If so, in 
what way? 

3. Has your daughter/son commented on the program? If so, describe comments. 

4. What is your opinion of the program? 
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Tips on Interviewing 

1. Ask friendly questions to help the person being interviewed feel more at ease. 

2. Restate questions when the interviewee looks puzzled. 

3. Express interest, periodically. 

4. Incorporate/repeat interviewee's key phrases and terms in questions and in notes. 

5. Express/reaffinn complexity of program requirements. 

6. Avoid a series of questions beginning with 'why' (may contain a judgmental 
component). 

7. Ask for a concrete example if response is vague (I wonder if you could give me an 
example of that?). 

8. When the interviewee looks uncomfortable talking about a particular topic, *'tread 
gently." 

9. Emphasize the sharing nature of the interview and avoid judgmental queries. 

10. Acknowledge any "ventilation" of feelings that occur before proceeding, to next 
question. 

1 1. If possible, try to take only brief notes during the interview itself and concentrate on 
the responses. 

12. Confirm/double check important points. 

13. Take a break during lengthy interviews. 

14. Jot down exact quotes or phrases once in a while. They add some personal flavor to 
notes. 

15. Allow only limited freedom for interviewee to go off on a "tangent." Maintain focus 
on improvement issues, rather than respondent (or your own) interests. 

16. At close of interview, allow for possibility to return to ask for any necessary 
clarification. 

Source: Spradley, J.S. (1979). The Ethnographic Interview. New York. Holt, Rinehart and 
Winston. 
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Conducting Observations 



1. Classroom observations are a very expensive and time-consuming approach to 
evaluation, so be sure that you can afford and/or really need this information. 

2. The determination of what behaviors, events, and environmental features you will be 
observing is essential before deciding to use this measure. 

3. The kind of training needed for observers must be carefully planned. 

4. All observers need training. 

5. Observers should be carefully selected. You need observers that are unobtrusive, 
preferably not familiar with teachers or students to be observed. 

6. All teachers are not amenable to having an observer in their classrooms. A meeting 
with teachers to explain evaluation and alleviate fears is recommended. If possible, 
teachers should volunteer for this activity. 

7. Be prepared for complications in scheduling, e.g., substitute teachers, shortened 
classes, school events preempting classes, students leaving on field trips. 
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Document Review Checklist 



Definition 

A "document" may be defined as any kind of written or printed information that can be used in 
support of or as evidence for a particular study. By collecting, sequencing, and interpreting 
documents, the evaluator can trace the origins of a given situation and put it in context. 

Types of Documents 

Documentary information can include: 

• school records 

• census figures 

• personnel files 

• socioeconomic facts 

• test scores 

• medical information 

• description of proposed program objectives, implementation plans, management plans, 
budget 

• demographic information 

• program philosophy 

• background material describing problem, content, and context 

• needs assessment information 

• infontiation about community, school, industry 

• materials, such as objectives, lesson plans, instructional guides, teachers' manuals, 
textbooks, handouts, testing instruments 

• progress reports, evaluation reports 

Western Regional Center for Drug-Free Schools and Communities_ 
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Planning for Gathering Documentary Information 

1. What are the documents that are relevant to the program being evaluated? 

2. How do you determuie the "relevant" documents? 

3. Whom would you ask to make this determination? 

4. Where are the locations at which these documents are stored? What are the hours these 
offices/locations are open for you to visit? Who knows this information? 

5. What are the names of the people who supervise/monitor/store these documents? Again, 
who knows this information? 

6. Whose permission do you need to examine these documents? 

7. Can the documents be removed from any of the locations or must they be reviewed at the 
site? Who can tell you this information? How long can you keep the documents? 

8. For each document or type of document what information are you looking for? 
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Access to Data Sources 



After determining the information you want to obtain, identifying the groups of people from 
whom you will be soliciting this information and the types of documents you will be reviewing, 
and matching with appropriate data collection instruments, then you will need to determine how 
to get to these people and documents for the information. In order to make this determination 
you will need to be informed of and act according to the following policies and procedures: 

L District/school policies for conducting an evaluation and established procedures for 
contact with board members, staff, students, parent, and community school-based 
organizations 

2. The legal restrictions and requirements for examining and reporting confidential 
information 

3. Any school/community protocol to be observed in the conducting of an evaluation. 

This information is essential to know in the early planning stage of your evaluation so that: 

1. You discover if you can implement your evaluation 

2. You can revise any changes needed in the evaluation design based on school/community 
guidelines/policies/restrictions 

3. You can build in enough time to carry out needed access tasks. 

Some examples of access issues are: 

1. K you plan to intep/iew students: 

Who needs to know and/or from whom do you request permission? teachers? principals? 
superintendent? parents? 

2. If you plan to examine student records: 

What are the district, the federal, and the state policies regarding access to, and reporting 
of, confidential information? 

Also, be aware of courtesies to be extended and attention to public relations in conducting an 
evaluation. Some people do not need to know about the study but your evaluation efforts would 
be well served if you provided this information to not only key members of the business 
community, but also to religious, civic, and other community organizations. 
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Selecting Instruments 

Constructing instruments is a very specialized and technical field requiring extensive training and 
experience. It is recommended that school districts check first to see what instruments are 
available be':ore attempting to develop their own. 

Scheirer and Rezmcvic (1983, p. 609) summarizes five measurement criteria for assessing the 
degree of implementation. They are: 

1) the use of multiple measurement techniques, 

2) the presence of an operational definition, 

3) the examination of reliability, 

4) the assessment of validity, and 

5) the use of sampling 

Using multiple measurement techniques 

Scheirer and Rezmovic (1983, p. 604) classified the techniques used to assess implementation in 
the following order based on the extent of human judgment required: 





Technical measures 
of equipment 
performance; 
technical measures 
of output 


Unobtrusive 
indicators 


Behavioral 
observations 


Institutional 
records; 
interviews; 
questionnaires 


Ethnographic 
observations 



They recommend that whenever possible several techniques should be used to assess 
implementation since it is a complex concept. Also, the use of multiple methods will decrease the 
biases and disadvantages of each individual technique. For example, responses from 
questionnaires can be supplemented with available institutional records or behavioral 
observations. 

Defining program implementation 

Scheirer and Rezmovic (1983) further points to the importance of specifying what one means by 
"implementation," that "an operational definition connects the empirical observations to the 
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underlying logic of the construct being measured, and thus clarifies the conceptualization of key 
concepts." With the presence of an operational definition, one can "proceed more efficiently, 
logically, and with greater clarity of purpose" (p, 612). 

An example which illustrates an operational definition of implementation is described in a journal 
article titled "Effects of Program Implementation on Adolescent Drug Use Behavior" by Pentz, et 
al (1990). In it, the authors defined the "quality" of implementation as (1) adherence (whether 
program was implemented or not ), (2) exposure (amount of program delivered to the 
participants), and (3) reinvention (the extent to which implementors deviate from the program 
standard). 

Examining reliability 

It is important to know if an instrument is reliable or not Reliability refers to the extent to which 
an instrument is free from errors of measuiement. In other words, if there is no change in the 
entity being measured, the results should be the same each time that it is measured. An instrument 
having high reliability is more apt to detect true changes than one that has low reliability. 
Instrument developers have primary responsibility for collecting and reporting evidence 
concerning tlie reliability of their instruments. 

Assessing validity 

Evidence of validity ensures that the instrument is measuring what it is supposed to measure. The 
traditional means of showing validity falls into these three categories: 

a) Consti"uct validity provides evidence that the results derived from the instrument 
indeed is a measurement of the theoretical construct (i.e., program 
implementation). Construct validity is usually assessed from a variety of sources 
such as intercorrelations among questionnaire items, close relationship of the 
instrument to other measures that assess the same construct, and analyses of 
individual responses to an instrument. Statistical procedures commonly used are 
factor analysis or a multitrait-multimethod matrix. 

b) Criterion validity demonstrates that the results gathered from the instruments are 
related to one or more outcome criteria of interest. According to the Standards 
for Educational and Psychological Testing (1985), criterion validity can be 
obtained through two means. One is by a predictive study and the other is by a 
concurrent study. A predictive study would be able to demonstrate the accuracy in 
which one specific component occurring early in the implementation process can 
predict later overall implementation. A concurrent study would collect prediction 
and criterion information at the same time, 

c) Content validity shows the extent to which the sample of items or questions on an 
instrument represent some specified universe or content domain. Expert 
judgments arc often used to make this determination. 
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Using sampling strategies 

The sampling strategy used in collecting data affects the adequacy of the information collected. 
For example, a school is interested in assessing teachers' attitudes toward their Student Assistance 
Program but only interviewed teachers who had made referrals. That would leave out the 
opinions of teachers who had nd made referrals. The results would certainly be biased toward a 
particular group of teachers. The type of sampling strategies commonly used are random sample, 
stratified sample, or a sample of convenience. The sample of convenience should be avoided as it 
introduces too many possibilities for bias and inaccuracy in results. 

Things to consider 

Before deciding on what instruments to use in evaluating program implementation, consider the 
foDowing points: 

1. What technical expertise in program evaluation do you have access to? 

2. What are the costs involved in purchasing instruments and using them to gather 
information? 



3. How can you improve on your program record keeping? 

4. How can you supplement self-reports of participants with more objective data? 

5. How will you use and disseminate the results of the information you have gathered about 
program implementation? 

6. How will your evaluation of program implementation dovetail with your evaluation of 
student outcomes? (Refer to the Western Regional Center's publication on Surveys of 
Student Alcohol and Other Drug Use: A Consumer's Guide, Second Edition (September 
1991) for suggestions on assessing student outcomes.) 

In the next section, a number of instruments have been summarized to give readers some idea of 
what instruments are currently available to evaluate program implementation. 
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Overview 



Sixteen instruments have been sunvnarized in this section. Eleven instruments are the paner-and- 
pencil variety such as questionnaires, checklists, or guides for face-to-face interviews. Three are 
inicrocomputer software programs and two describe program record-keeping forms. 

All the instruments reviewed have been developed by researchers or practitioners in the last nine 
years and have been used in a school setting in the U.S. Some of the instruments are targeted at 
specific groups such as administrators, classroom teachers, or students. Others are developed for 
use by school-community teams or a combination of representatives from various groups. 

Information on reliability of the instruments has generally not been reported. Evidence of validity 
offered is usually that of content validity. It is apparent that instruments on program 
implementation are still at an early stage of development with more time needed for refinements. 
Nevertheless, these instruments provide a sampling of tools that can provide useful information on 
AOD program implementation. 

The authors have expressed their willingness to share their instruments and either do not charge 
for the instruments or may ask for a modest fee. The contact persons or agencies listed at the end 
of each abstract should be contacted directly if more information is needed. 
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Administrator Questionnaire 



Author/Agency: Ray Tricker, Department of Public Health, Oregon State University 
Year Developed: 1985 
Copyrighted: Yes 

Possible Uses: The questionnaire can be used to assess the attitudes and perceptions of 
administrators toward the AOD curriculum being used in their schools or districts. 

Cost: None 
Description: 

A* Target Group: Administrators or AOD coordinators in schools or districts using an 
AOD curriculum 

Areas Assessed: Perceptions of the current situations in the schools or districts and the 
way the respondent would like them to be with respect to: 

• Cost to implement the curriculum 

• The coordinator 

• School and district administrators 

• Teachers 

• Teacher inservice 

• The curriculum 

• The community and parents 

• Tlie evaluation of the implementation process 

C. Format/Scoring: The questionnaire contains 30 questions; each question is answered on 
a two-part Likert scale as shown in the sample question below: 
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The top half of the response scale represents the way the situation is perceived by the 
respondent and the bouom half represents the way the respondent would like it to be. The 
discrepancies between the two parts of the scale could be easily detemnined and the 
differences rank ordered from low to high, 

D. Time to Complete: Less than 20 minutes 

£• Instrument Development: Adapted from an administrative implementation perceptions 
questionnaire used in previous research, this particular instrument was used in a mailed 
survey sent to district superintendents, curriculum coordinators, school principals and 
program coordinators in three Oregon school districts in 1985. It was part of a doctoral 
dissertation study evaluating the implementation of AOD prevention curriculum. 

Available From: 

Dr. Ray Tricker, Assistant Professor 

Oregon State University, Department of Public Health, 

Waldo Hall, Room 311, Corvallis, OR 97331-6406. 
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The Community Drug and Alcohol Assessment Guide for Youth 



Author/Agency: David S. Anderson and Steven M. Janosik, Integrated S:^bstance Abuse 
Consultations 

Year Developed: 1988 

Copyrighted: Yes 

Possible Uses: Results from the Assessment Guide can be used as a needs assessment and for 
improvement of prevention efforts across community groups. It can also be used as a tool to 
develop a coordinated community effort in prevention. 

Cost: $14,50. Multiple copies ordered thereafter are $2.00 each. Communities may copy their 
originals for use m their projects. 

Description: 

A. Target Group: Community leaders 

B. Areas Assessed: The extent of involvement in alcohol and other drug prevention by the 
groups listed below: 

• Community at large 

• Health services 

• Social services 

• Schools 

• Religious institutions 

• Parents 

• Civic groups 

n Business and industry 

• Governmental agencies 

• Courts 

• Law enforcement agencies 

• Media groups 

• Restaurants and other entertainment businesses 

C. Format/Scoring: The Assessment Guide consists of three sections: an 8-page 
questionnaire of 144 questions with columns for checking "yes" or "no" responses, a score 
sheet, and a chart to plot the percentages of affirmative responses. The results reflect the 
strength of prevention efforts within each of the various groups within the community. 
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D. Time to Complete: About 20 to 30 minutes 

E. Instrument Development: Inspiration for the guide came from a school superintendent 
who suggested to the authors that they develop an instrument to assess prevention efforts 
beyond the schools. The 12 groups listed above were identified from a review of the 
literature. 

Available From: 

Dn David S. Anderson 
Integrated Substance Abuse Consultations 
P. O. Box 7505, Arlington, VA 22207 
Telephone: (703) 237-3840. 
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Level of Implementation Profile 



Author/Agency: Western Regional Center for Drug-Free Schools and Communities, Northwest 
Regional Educational Laboratory 

Year Developed: 1987 

Copyrighted: Yes 

Possible Uses: To monitor the extent of local implementation of key components of a 
comprehensive AOD program. It is most helpful when used on a statewide basis and 
supplemented with more objective measures. 

Cost: None 

Description: 

A. Target Group: School, district, or community team 

B. Areas Assessed: 

• Needs assessment 

• Action plan 

• Policies and procedures 

• Prevention 

• Intervention 

• Support/Aftercare 

• Evaluation 

• Dissemination 

• Parent/Community involvement 

• Student involvement 

• Need for training and technical assistance 

C. Format/Scoring: The questionnaire consists of items which ask the respondents to circle 
the stage of implementation that best describes where their programs are on each of the 
components listed in "B" above. The five stages of implementation are: 

1. No implementation 

2. Initial planning/discussion but no implementation yet 

3. Eily implementation 

4. Complete implementation 

5. Institutionalization 
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In addition, the questionnaire asks respondents to indicate their need (high, medium, or 
low) for training and technical assistance in 16 content areas. 

D. Time to Complete: About fifteen minutes 

E. Instrument Development: The questionnaire was developed initially by Western 
Regional Center staff to collect information for reporting required by the funding source. 
The stages of implementation used were derived from the literature on program 
implementation and stages of educational change. In recent years, the Hawaii State 
Department of Education has requested all public schools in Hawaii to complete this 
questionnaire at the end of each school year as part of their statewide planning and 
evaluation activities. 

Available from: 

Judith A. Johnson, Director 

Western Regional Center for Drug-Free Schools and Communities 
Northwest Regional Educational Laboratory 
101 S.W. Main Street, Suite 500 
Portland, OR 97204 

Telephone: (503) 275-9476 or 1-800-547-6339, ext. 476. 
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Parent's Self Test: A Guide for Helping Parents 
Assess Their Own Drug Use 



Author/Agency: John K. Delaine, Wisconsin Clearinghouse, University of Wisconsin - Madison 
Year Developed: 1985 
Copyrighted: Yes 

Possible Uses: The questionnaire is to help individuals assess their attitudes and own use of 
drugs as well as possible reasons for their behavior. The instrument recommends that parents 
contact a local agency for help if they suspect they have a problem with alcohol and/or other 
drugs. 

Cost: 50 for $10; 200 for $34; 500 for $70; 1000 for $120 
Description: 

A- Target Group: Parents 

B, Areas Assessed: Drug use habits regarding 

• Alcohol 

• Over-the-counter and/or prescription medication 

• Cigarettes 

• Marijuana 



C. Format/Scoring: The questionnaire has 22 items with "yes" or "no" check-off responses 
which can be easily tallied. 

D. Time to Complete: 5 minutes 

E. Instrument Development: It is taken in part from the book Who*s Raising the Family 
(1981) by John K. Delaine, Wisconsin Clearinghouse. 
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Available From: 

Wisconsin Clearinghouse 
University of Wisconsin-Madison 
P.O. Box 1468 
Madison, WI 53701-1468 
Telephone: 800-322-1468 
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Personal Interview Questionnaire for Teachers 



Author/Agency: Ray Tricker, Department of Public Health, Oregon State University 
Year Developed: 1985 
Copyrighted: Yes 

Possible Uses: Can be used to assess teachers' attitudes and the degree of implementation of an 
AOD curriculum in a school or district. 

Cost: None 

Description: 

A. Target Group: Classroom teachers using an AOD curriculum 

B. Areas Assessed: 

i» Teaching and prepararion time 

^ Quality of inservice training 

• Attitudes toward the curriculum 

• Administrative support 

• Compliance with the curriculum guidelines 

• Evaluation of implementation 

• Community involvement 

• Teacher perceptions of the implementation procedures 

C. Format: The questionnaire is for use in onsite interviews. It consists of 56 questions 
covering the eight areas listed in "B" above. Some of the questions require only brief 
answers while others ask for detailed responses, 

D. Time to Complete: 45-60 minutes 

E. Instrument Development: The interview questionnaire was developed from information 
gathered during visits to three Oregon school districts in 1985 and reviewed for content 
validity by a panel of health and drug education experts at the University of Oregon, The 
project was part of a doctoral dissertation study examining the financial costs and teacher 
attitudes toward the implementation of the Here's Looking at You, Two and the Starting 
Early curricula. Program coordinators and classroom teachers in 21 different Oregon 
public schools were interviewed. 
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Available From: 

Dn Ray Tricker, Assistant Professor 

Oregon State Universit>', Department of Public Health 

Waldo Hall 311, CorvaUis, OR 97331-6406. 
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SAP: Participants Rating Sheet for Group Activity 



Author/Agency: M. Amos Clifford and Mary Davis, National Organization of Student 
Assistance Programs and Partners 

Year DeveKoped: 1990, finalized 1991 

Copyrighted: Yes 

Possible Uses: To elicit feedback from support group or peer group participants to develop more 
cohesiveness within the group and to improve group facilitation. Results of the questionnaire can 
also be used for ongoing planning as well as to document and publicize program efforts to school 
or district administration, school board, students, parents, and other school staff. 

Cost: $39.50 plus $3.50 for shipping and handling. This cost is for the manual Evaluation Tools 
for Student Assistance Programs which contains this instrument and other evaluation instruments 
and resources. 

Description: 

A. Target Group: Support group or peer group participants 

B. Areas Assessed: 

• Opinions about program strengths 

• Opinions about support group or peer ^.roup weaknesses 

• Recommendations for support group or peer group improvement 

• Satisfaction with confidentiality issues 

• Satisfaction with level of trust among group members 

• Satisfaction with group interaction and communication 

« Satisfaction with helpfulness of support group or peer group 

C. Format/Scoring: The instrument is three pages and can be used in group settings. It 
begins with a protocol that summarizes the purpose of the form and then describes how to 
administer, score and interpret results. The rating sheet contains 22 statements which the 
participant rates as to how often each of them is true about the group, followed by four 
open-ended questions. Responses can be tallied to determine program strengths and 
weaknesses, 

D. Time to Complete: 10 to 15 minutes. 
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E. Instrument Development. The instrument was developed by staff at the National 
Organization of Student Assistance Programs and Partners (NOSAPP) as part of an 
evaluation manual to assist school districts in evaluating their student assistance programs. 
Many of the instruments in the manual have been pilot-tested in a number of school 
districts and feedback on the contents has been solicited from contacts throughout the 
countr>^ 

Available From: 

National Organization of Student Assistance Programs and Partners 
4760 Walnut, Suite 106 
Boulder, Colorado 80301 
Telephone: 1-800-972-4636. 
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SAP: Participant Satisfaction Survey 



Author/Agency: M. Amos Clifford and Mary Davis, National Organization of Student 
Assistance Programs and Partners 

Year Developed: 1990, Finalized 1991 

Copyrighted: Yes 

Possible Uses: To assess participant's attitudes and level of satisfaction with the student 
assistance program so that staff can make the necessary changes when needed. Results of the 
questionnaire can also be used for ongoing planning and to document as well as to publicize 
program efforts to school or district administration, school board, students, parents, and other 
school staff. 

Cost: $39.50 plus $3.50 for shipping and handling. This cost is for the manual. Evaluation Tools 
for Student Assistance Programs, which contains this instrument and other evaluation instruments 
and resources. 

Description 

A. Target Group: Student assistance program participants 

B. Areas Assessed: 

• Opinions about program strengths 

• Opinions about program weaknesses 

• Recommendations for program improvement 

• Satisfaction with confidentiality issues 

• Satisfaction with staff expertise 

• Satisfaction with personal participation 

C. Format/Scoring: The instrument is three pages and can be used in either group settings 
or with individuals. It begins with a protocol that summarizes the purpose of the form and 
then describes how to administer, score and interpret results. The second page contains 
four open-ended questions asking for opinions about the student assistance program. The 
third page lists 16 statements which participants can respond to by indicating on a fwe- 
point scale the extent of their agreement or disagreement with each statement. Students' 
ratings on each statement can be averaged, with low average scores suggesting areas that 
may need improvement. 

D- Time to Complete: 10 to 15 minutes. 
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£• Instrument Development. The instrument was developed by staff at the National 
Organization of Student Assistance Programs and Partners (NOSAPP) as part of an 
evaluation manual to assist school districts in evaluating their student assistance programs. 
Many of the instruments in the manual have been pilot-tested in a number of school 
districts and feedback on the contents has been solicited from contacts throughout the 
country. 

Available From: 

National Organization of Student Assistance Programs and Partners 
4760 Walnut, Suite 106 
Braider, Colorado 80301 
Telephone: 1-800-972-4636. 
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SAP: Staff Satisfaction Survey 



Author/Agency: M. Amos Clifford and Mary Davis, National Organization of Student 
Assistance Programs and Partners 

Year Developed: 1990, finalized 1991 

Copyrighted: Yes 

Possible Uses: To assess staffs attitudes and level of satisfaction with the student assistance 
program for ongoing planning and improvement of services. Results of the questionnaire can also 
be used to document and to publicize program efforts to school or district administration, school 
board, students, parents, and other school staff. 

Cost: $39.50 plus $3.50 for shipping and handling. This cost is for the manual. Evaluation Tools 
for Student Assistance Programs, which contains this instrument and other evaluation instruments 
and resources. 

Description: 

A. Target Group: School staff and community representatives who work with the student 
assistance program 

B. Areas Assessed: 

• Opinions about program strengths 

• Opinions about program weaknesses 

• Recommendations for program improvement 

• Satisfaction with confidentiality issues 

• Satisfaction with student assistance staff expertise 

© Satisfaction with student assistance staffs use of other resources 

• Responsiveness of student assistance staff to their concerns 

• Usefulness of services provided by the student assistance program 
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C. Format/Scoring: The instrument is thtree pages and can be used in either group settings 
or with individuals. It begins with a protocol that summarizes the purpose of the form and 
then describes how to administer, score and interpret results. The second page contains 
six open-ended questions asking for opinions about the student assistance program. The 
third page lists 16 statements which respondents can respond to by indicating on a five- 
point scale the extent of their agreement or disagreement with each statement Ratings on 
each statement can be averaged, with low average scores suggesting areas that may need 
improvement 

D. Time to Complete: 10 to 15 minutes. 

E. Instrument Development The instrument was developed by staff at the National 
Organization of Student Assistance Programs and Partners (NOSAPP) as part of an 
evaluation manual to assist school districts in evaluating their student assistance programs. 
Many of the instruments in the manual have been pilot-tested in a number of school 
districts and feedback on the contents has been solicited from contacts throughout the 
country. 

Available From: 

National Organization of Student Assistance Programs and Partners 
4760 Walnut, Suite 106 
Boulder, Colorado 80301 
Telephone: 1-800-972-4636. 
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The Secondary School Drug and Alcohol Assessment Guide 



Author/Agency: Steven M. Janosik and David S. Anderson, Integrated Substance Abuse 
Consultations 

Year Developed: 1987 

Copyrighted: Yes 

Possible Uses: To assess and to identify strengths and weaknesses in school prevention efforts 
for program development and improvement. 

Cost: $7.50 per copy ($6.00 per copy for 10 or more; $5.00 per copy for 50 or more). 
Description: 

A, Target Group: The instrument is targeted primarily at secondary school administrators. 
The guide recommends that a variety of individuals such as teachers, school board 
iTsembers, parents, and suidents participate in completing the instrument. 

Areas Assessed: The extent to which the school is addressing AOD prevention in these 
areas: 

• Policy 

• Procedures 

• School functions and parties 

• Training for teachers and staff 

• Education for club advisors 

• Education for students 

• Education for parents 

• Access to treatment and intervention 
« Reseaich 

9 Community involvement and leadership 

C. Format/Scoring: The Assessment Guide consists of 118 yes/no questions. No score 
sheet is included, but the "No" answers could be prioritized and compared across the 
different components being assessed. 

D. Time to Complete: About 20 to 30 minutes 
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Instrument Development: The authors were interested in developing a Jc^ cost, simple 
tool for educators to use in assessing the extent to which their schools were addressing 
prevention and intervention strategies. According to the authors, a particularly useful 
feature of the instrument is that it allows for different groups of people within the school 
setting to compare their perceptions of each of the program components. A review of the 
literature provided information about important features of AOD programs. 

Available From: 

Dr. David S. Anderson 
Integrated Substance Abuse Consultations 
P.O. Box 7505, Arlington, VA 22207 
Telephone: (703) 237-3840. 
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Self- Assessment Instrument for School Programs 
in Alcohol and Other Drug Prevention 



Author/Agency: Western Regional Center for Drug-Free Schools and Communities, Northwest 
Regional Educational Laboratory 

Year Developed: 1988 

Copyrighted: Yes 

Possible Uses: A simple, informal tool to assist districts in reviewing and identifying areas for 
additional development in their AOD program. It can facilitate staff participation in goal-setting 
and stimulate learning, reflection and debate. The results should be supplemented with more 
objective measures whenever possible as self-reports may not always be accurate. 

Cost: None 

Description: 

A. Target Group: All members of a school or district team, including teachers, counselors, 
principals, and central office administrators 

B. Areas Assessed: 

• School team involvement 

• Parent/community involvement 

• Needs assessment 

• Program planning 

• Policy development 

• Implementation of prevention curriculum 
« implementation of assistance programs 

« Evaluation 

• Dissemination 

C. Format/Scoring: The instrument consists of 34 statements referring to a series of 
indicators about the program components listed above. Each statement can be rated on a 
continuum from "1" (not at all like our program) to "5" (very much like our program). 
Ratings may be totaled or averaged for each component to provide an indication of 
specific program practices that need improvement or greater emphasis. 
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D. Time to Complete: Ten to fifteen minutes for each team member to individually 
complete the instrument and another fifteen minutes to share results, discuss perceptions, 
and select target areas for improvement. 

E. Instrument Development: The instrument was developed for use with participants at 
Western Regional Center workshops. Used many times in the past, the tool has often 
stimulated lively discussions among team members and encouraged broad participation in 
setting priorities. 

Available From: 

Judith A. Johnson, Director 

Western Regional Center for Drug-Free Schools and Communities 
Northwest Regional Educational Laboratory 
101 S.W. Main Street, Suite 500 
Portland, OR 97204 

Telephone: (503) 275-9476 or 1-800-547-6339, ext. 476. 
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What Do You Think About the Drug Abuse Problem 
and Drug Education? 



Author/Agency: Daniel M Mayton n and Liza Nagel of Lewis & Clark State College 
Year Developed: 1991 
Copyrighted: Ko 

Possible Uses: Assess teachers' attitudes to help policy makers in understanding what motivates 
teachers to comply with federal mandates. Identifies barriers to teachers' provision of drug 
education. 

Cost: None 

Description: 

A. Target Group: Teachers 

B* Areas Assessed: Teachers' attitudes, intentions and behaviors regarding 

• Severity of drug abuse problems nationally 

• Severity of drug abuse problems locally 

• Efficacy of drug education and drug infusion strategies 

• Self efficacy in implementing drug education and drug education infusion strategies 

• Infusion strategies 

• Attitude toward drug education 

• Subjective norms regarding drug education 

• Motivations to comply 

• Intentions to implement drug education 

• Intentions to learn more about drug education 

• Current use of drug education strategies in the classroom 

• Demographic information 

C* Format/Scoring: The questionnaire consists of 37 statements, with response options on 
a continuum from "strongly agree" to "strongly disagree." Responses to the two-page 
questionnaire can be summarized to determine how many teachers agree or disagree with 
each statement. 

D, Time to Complete: 20 minutes 
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E. Instrument Development: The questionnaire was developed by the authors as part of a 
research study funded by the U.S. Department of Education in 1991 to assess the attitudes 
of certified teachers toward drug education in rural north Idaho. The questionnaire was 
used in a mail survey sent to over 1,000 teachers during February 1991 and 1992. The 
items in the questionnaire were based on two social psychological theories of attitude 
formation and maintenance useful in predicting health-related behavioral change. 

Available From: 

Dr. Liza Nagel 

Dnig Education Program 

Lewis-Clark State College 

301 Spalding Hall, 500 8th Avenue 

Lewiston,ID 83501. 
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□ 

Drug Information, Assessment and Decisions for SchooIs*DI ADS 



Author/Agency: Dr. Kris Bosworth and Richard Yoast, University of Indiana; Doug White and 
Deborah Coe-Bradish, Wisconsin Clearinghouse, University of Wisconsin-Madison. 

Year DeveJoped: 1990 

Copyrighted: Yes 

Possible Uses: DIADS can be used as a tool to assess user's prevention programs and identify 
areas for improvement, to help in program planning, to locate new instructional resources, and for 
training. 

Cost: $189.95 per copy plus $5 per package for shipping and handling; $759.95 for five copies. 
Description: 

A. Target Group: Administrators and AOD Coordinators 

Areas Assessed: 

• Curriculum 

• Support of community, administrators, staff and students 

• Coordination with community 

• Drug policy or assistance program 

C. Format: DiADS is an interactive software program. It allows users to respond to each 
of 26 questions to determine whether various components are being implemented or not 
The responses will be used to assess the chances for program success. The users then 
have the option to change their responses to see how this would change the probability for 
success. In addition, a feedback matrix will indicate program's strengths and weaknesses 
based on users' replies and provide data comparing these replies to the average responses 
of all parties who have used DIADS at the site. 

D* Time to Complete: 10-30 minutes for an individual, or a team might spend an hour or 
more. It depends on how much people know about the program and how many use it at 
once. 
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E. Instrument Development: Over 300 experts in the field were polled to detemiine 

agreed-upon characteristics of successful programs. The list was prioritized and validated 
by experts in schools before it was used to generate questions in DIADS. The experts 
agreed with DIADS' probability assessment 92% of the time. 



Hardware Requirements: Requires an IBM PC or true compatible computer with 512K 
of memory using a CGA, EGA or VGA graphics card with a disk drive for 5-1/4 inch, 3- 
1/2 inch, 360K disks using DOS 2.0 or higher. An Apple version is also available. 



Available From: 

Wisconsin Qearinghouse 
University of Wisconsin-Madison 
P.O. Box 1468 
Madison, WI 53701-1468. 
Telephone: 1-800-322-1468 
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Evaluation and Quality Improvement for Prevention (EQUIP) 



Author/Agency: Eric Goplenid, Office for Substance Abuse Prevention 
Year Developed: 1991, ongoing revision 
Copyrighted: Yes 

Possible Uses: The program helps prevention coordinators to identify strengths and weaknesses 
in their current programs, increased their awareness of existing effective programs; and assists in 
determining evaluation instruments and procedures that would be most appropriate to the user's 
program. 

Description: 

A. Target Group: Primarily district and state administrators, consultants, and community 
coalition personnel. May also be used by building personnel. Depends upon degree of 
comfort with computers. 

B. Areas Assessed: 

• Evaluability of AOD programs 

• Selection of AOD instruments (The program does not assess instruments per se but 
guides selection.) 

C Format: EQUIP is a microcomputer software program on a 3-1/2" floppy disk that 
provides evaluability assessment, information on instruments, evaluation guidelines, and 
best prevention practices. Each section will guide users through a set of questions. Based 
on the responses, the expert system will present users with tips and recommendations for 
further action. The evaluability assessment asks up to 14 questions of the user and then 
provides advice on-screen based upon the profile generated by those answers. The 
instrument selection section also asks up to 14 questions and then lists instruments and 
sample questions that would be appropriate based on the answers. 

D. Time to Complete: Probably requires at least fifteen minutes to become familiar with the 
program, and perhaps an hour to completely work through the evaluability assessment and 
the section on selecting instruments. (More tiume would be needed to peruse section on 
best practices.) 
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E. Instrument Development: The expert system program was designed to help users 
improve the quality of their prevention services without the need to spend valuable time 
on literature searches. A combination of literature review and the author's personal 
experience evaluating AOD programs led to the design of the program. The section on 
selection of instruments focuses heavily on reliability and validity of instruments. EQUIP 
is still being revised. 

F. HardM'are Requirements: Requires DOS-based machine, IBM or IBM compatible with 
a 3-1/2" disk drive and 700K of memory. 

Available From: 

Dr. Eric Goplerud 

Office for Substance Abuse Prevention 
5600 Fishers Lane, Rockwall 1 1 
RockviUcMD 20857 
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SAP Assistant 



Author/Agency: Western Regional Center for Drug-Free Schools and Communities, Northwest 
Regir lal Educational Laboratory 

Year Developed: 1989, revised 1992 

Copyrighted: Yes 

Possible Uses: Designed to facilitate program management and reporting of Student Assistance 
Programs. The system can be used to track both students served by the program and school staff 
providing services. 

Cost: A master copy and two days of onsite technical assistance are available at no charge for 
schools in the Western region. For those districts interested in a data collection subscription 
service offered by the Northwest Regional Educational Laboratory, there is a fee to store, 
tabulate, and summarize data to be entered from optical scan forms. 

Description: 

A. Target Group: SAP personnel in moderate to large school districts 

B, Data Entry Forms to Record: 

• Program descriptions 

• Schools participating in program 

• Student information (history of referrals, services received, and indicators of 
progress) 

» Inservicing and presentations made by staff on the SAP 

• Staff information (their roles and training received) 

C» Format: The SAP Assistant is a microcomputer database system with operations guided 
by menus and pop-up help windows. The main menu provides access to data entry, data 
management, and database maintenance. Many flexible reports may be accessed through 
menus to respond to typical data needs. A quick-report feature allows the user to create 
custom reports to answer ad hoc requests for data. 
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D. Time to Complete: Implementing a computerized tracking system such as the SAP 
Assistant takes time. For districts with experience in computerized record-keeping, it will 
still require about six months to train personnel, to set up procedures for a district-wide 
data collection system, to enter the necessary data, and finally have a working system in 
place, 

E. Instrument Development: The SAP Assistant was developed using a commercially 
available relational database management software - DataBase ~ to minimize development 
time and facilitate local enhancements or adaptations. The first local site that used the 
program was the Portland Public Schools (Oregon) in 1988, Since then, other school 
districts such as the Boise School District in Idaho have installed the SAP Assistant to 
longitudinally track students and staff involved in their S APs, A new feature added in 
1992 is a data collection subscription service to assist school districts who have neither the 
time nor technical resources to enter, tabulate, and summarize their own data. For a set 
fee depending on the amount of data involved, the Northwest Regional Educational 
Laboratoiy will process optical scan forms (to be completed by district personnel) and 
summarize the data for the district on a monthly cycle, 

F. Technical Information: The SAP Assistant requires an IBM-compatible microcomputer 
(286 or higher recommended) with at least a 40 megabyte hard disk drive, 640K of 
memory (560 of which is free), MS DOS 3,1 or higher, and a printer. The relational 
database management software DataEase is produced by DataBase International, 

Available From: 

Judith A, Johnson, Director 

Western Regional Center for Drug-Free Schools and Communities 
Northwest Regional Educational Laboratory 
101 S,W, Main Street, Suite 500 
Portland, OR 97204 

Telephone: (503) 275-9476 or 1-800-547-6339, ext, 476, 
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JJ Student Assistance Program Recordkeeping Forms 

and Student Questionnaires 

Author/Agency: D. Paul Moberg, Center for Health Policy and Program Evaluation, University 
of Wisconsin-Madison 

Year: Developed 1983, Revised February 1984 
Copyrighted: No 

Possible Uses: A set of six forms for documentation, tracking, and evaluation of student 
assistance program activities for use by student assistance programs and external evaluators. 

Cost: $3.25 per set for duplicating and mailing 

Description: 

A. Targei Group: Staff and clients of student assistance programs 

B. Areas Assessed: 

• Student Assistance Program Confidential Referral Form (3 pages). This form is to be 
used for each referral and provides sections where staff can circle the source of 
referral, the reasons for referral, and staff response to referral. 

• Student Assistance Program Confidential Initial I nterview Record (4 pages.) This 
form is to be used in documenting the screening of the student's problem and 
recording staff recommendations. 

• Student Ass istance Tracking Form. Service Record, and End of Year Assessment f3 
pages). This form is to be completed for each student who wu: involved in the 
student assistance program. It can be used to document the services provided, to 
track any changes in the student*s problem status during the year, and to record a 
year-end rating on the student by a Student Assistance staff most knowledgeable 
about the student. 

• Student Assistance Program Confidential Student School Performance Data (1 page). 
This form is to record the attendance, grade point average, disciplinary referrals, and 
suspensions on each student involved with the student assistance program. The 
information can be collected from school records to reflect the status before, during, 
and after the student's involvement in the student assistance program. 
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• Confidential Student Questionnaire (1 pages). This form is to be completed 
anonymously by the student and sent directly to an external evaluator. The 
questionnaire asks the student to respond to their use of alcohol and other drugs, 
AOD use among family and friends, student attitudes toward AOD use, and student 
assessment of the student assistance program that she/he has been attending. 
Included are the Mayer and Filstead's Adolescent Alcohol Involvement Scale (AAIS) 
and the Rosenberg Self-esteem Scale. 

• Follow-up Confidential Student Questionnaire (8 pages). This form is to be 
completed by each student who has been involved in student assistance services for at 
least three months by the end of the year. It parallels the Confidential Student 
Questionnaire described immediately above. The only difference is that ten follow-up 
questions have been added to ask about current friends' use of AOD, student 
participation in alternative leisure activities, student abstinence from AOD, and 
student reflection on past AOD use. 

C. Format: The questionnaires and recordkeeping forms have been developed so that when 
completed, they can be entered into a computer database for tabulation and data analysis. 

D. Time to Complete: Estimated staff time ranges from 5 minutes to 15 minutes per form. 
Estimated student time for questionnaires ranges from 15 to 25 minutes. 

E. Instrument Development: The recordkeeping forms and student questionnaires were 
developed by the author as part of an evaluation study of student assistance programs at 
two urban school districts in Wisconsin. Student assistance staff reviewed the forms and 
provided input for revisions which were incorporated by the author. Results of the study 
were presented at the 1988 annual meeting of tiie American Public Health Association in 
Boston, Massachusetts. Initial funding for the evaluation was provided by the Wisconsin 
Department of Public Instruction. 

Available From: 

Dr. D. Paul Moberg, Associate Dh-ector 
Center for Health Policy and Program Evaluation 
University of Wisconsin-Madison 
707 WARE Building, 610 Walnut Street 
Madison, WI 53705-2397. 
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= Jj Western Regional Center Student Assistance Program 

Referral Record 

Author/Agency: Western Regional Center for Drug-Free Schools and Communities, Northwest 
Regional Educational laboratory 

Year Developed: 1988 

Copyrighted: Yes 

Possible Uses: To track and monitor services provided to individual students, and to document 
the educational performance of students referred to the program. 

Cost: None 

Description: 

A, Target Group: SAP personnel 

B, Areas Assessed: For record keeping on 

• Student identification 

• Referrals 

• Pre-assessment services 

• Assessments 

• Post-assessment services 

• Educational indicators 

C, Format/Scoring: The referral record form is three pages long with spaces for recording 
information by hand or with a typewriter. The instructions are four pages long and explain 
how to complete each of the sections listed above. 

D, Time to Complete: Five minutes to write or type information on each client 

E, Instrument Development: This referral form has been used in Western Regional Center 
training on program evaluation in the past. It is part of a Student Assistance Program 
evaluation packet to assist school districts in setting up a carefully thought out record 
keeping system that places minimal burden on school staff. The materials were adapted 
from the work of SAP coordinators in the Pacific Northwest and might require some 
modifications to meet local situations. 
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Available From: 

Judith A. Johnson, Director 

Western Regional Center for Drug-Free Schools and Communities 
Northwest Regional Educational Laboratory 
101 S.W. Main Street, Suite 500 
Portland, OR 97204 

Telephone: (503) 275-9476 or 1-800-547-6339, ext. 476. 



/ 
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Overview 

This section provides additional readings and resources. It contains: 

1. A list of evaluation manuals and handbooks; 

2. A list of journals where articles and reports on alcohol and other drug programs may 
be found; 

3. A list of journals where articles and reports on evaluation may be found; 

4. Addresses and telephone numbers of resource centers available to educators in the 
Western region. 
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Evaluation Manuals and Handbooks 



The list below provides some publications that are easily available, inexpensive or free, and 
targeted to individuals with litde or no background in evaluation. 



1. Clifford, M.A., & Davis, M. (1991). Evaluation Tools for Student Assistance Programs. 
Boulder, CO: National Organization of Student Assistance Programs and Partners. 

The Evaluation Tools for Student Assistance Programs manual discusses what needs to be 
considered before starting an evaluation; Usts the steps involved in evaluating SAPs; and includes 
21 evaluation tools. Sections discussing purpose, administration protocol and results 
interpretation are included for each instrument. 



2. Hawkins, J.D., & Nederhood, B. (1987). Handbook for Evaluating Drug and Alcokol 
Prevention Programs, Rockville, MD: U.S. Department of Health and Human Services. 

This handbook offers a step -by-step approach for the non-expert to evaluate school or 
community-based prevention programs. Each step offers specific details and offers examples 
from school or community programs. The evaluation model suggests that tliere are evaluation 
questions that need to be asked about effort, effectiveness, and efficiency of the program. Once 
the evaluation questions are determined, evaluation design, identifying outcome measures, 
organizing and collecting the data, elementary data analysis, and reporting the findings foUow. 
Each topic is represented in the chapters (steps). 



3. Herman, J.L. (Ed.). (1987). Program Evaluation Kit. Newbury Park, CA: SAGE 
Publications. 

This series includes nine handy paperback books as follows: Evaluator's Handbook, How to 
Focus an Evaluation, How to Design a Program Evaluation, How to Use Qualitative Methods in 
Evaluation, How to Assess Program Implementation, How to Measure Attitudes, How to 
Measure Performance and Use Tests, How to Analyze Data and How to Communicate 
Evaluation Findings. A step-by-step approach with illustrations is used in each of the handbooks. 



4. lOX Assessment Associates. (1988). Program Evaluation Handbook: Alcohol Abuse 
Education; Program Evaluation Handbook: Drug Abuse Education; Program 
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Evaluation Handbook: Smoking Abuse Education. Los Angeles, CA: lOX Assessment 
Associates. 

These are a series of three handbooks developed for the Center for Health Promotion and 
Education, Office of Disease Prevention and Health Promotion, U.S. Department of Health and 
Human Services. The handbooks are based on preliminary goals or standards of behavior, 
knowledge, skill, and affective outcomes in each content area. Panels of experts identified 
important outcomes for each of the content areas (alcohol education, drug abuse education, and 
smoking education). Based on these outcomes, a series of criterion-referenced instruments was 
developed and included in each manual. These instruments are considered useful for adults and 
school-aged children. 

Each manual begins with an overview of the content area. A chapter on evaluation essentials 
provides the novice with an overview of the evaluation issues. The bulk of the manual is filled 
with brief assessment instruments. While the instruments appear to have great face validity and 
sometimes mirror questions used in national surveys, the reader is cautioned that the instruments 
have only been pilot-tested in small-scale tryouts and "have not been subjected to a formal 
empirical appraisal of their technical adequacy." 



5. Joint Committee on Standards for Educational Evaluation. (1981). Standards for 

Evaluations of Educational Programs, Projects, and Materials, New York: McGraw- 
Hill. 

A joint committee representing the American Educational Research Association, the American 
Psychological Association, and the National Council on Measurement in Education prepared 
thirty evaluation standards. The standards represent issues such as deciding whether you are 
ready to evaluate, the ethics surrounding evaluations, defining the evaluation problem, designing 
the evaluation, budgeting the evaluation, collecting and analyzing data, and reporting the results* 
Each standard is presented, followed by an overview, guidelines, pitfalls, caveats, an illustrative 
case, and an analysis of the case. It is an excellent overview for the beginning evaluator. 



6. Linney, J. A., & Wandersman, A. (1991). Prevention Plus III. Rockville, MD: U.S. 
Department of Health and Human Services. 

This is the third in a series of Prevention Plus publications from the Office for Substance Abuse 
Prevention (OSAP). It offers "up-to-date tools and techniques for alcohol and other drug (AOD) 
prevention planning and implementation." Both outcome and process evaluations are discussed 
and tools are provided for both. 
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Journals on Alcohol and Other Drug Prevention and Education 



There are no journals yet dedicated to dissemination of evaluation reports in the AOD field, so 
articles on evaluating AOD programs may appear in any of these journals: 



Addictive Behaviors (6/year) 
Pergamon Press, Inc. 
Journals Division 
Maxwell House 
Fairview Park 
Elmsford,NY 10523 
TEL: 914-592-7700 

American Journal of Drug and 
Alcohol Abuse (4/year) 
Marcel Dekker Journals 
270 Madison Avenue 
New York, NY 100 J 6 
TEL: 212-696-9000 

American Journal of Orthopsychiatry 
(quarterly) 

American Orthopsychiatric Association, 
Inc» 

19 W. 44th St. 

New York, NY 10036 

TEL: 212-354-5770 

American Journal of Public Health 

(monthly) 

American Public Health Association 
1015 15th St., NW 
Washington, D.C. 2005 
TEL: 202-789-5600 

Health Education Quarterly 

(quarterly) 

John Wiley & Sons, Inc. 

Periodicals Division 

605 Third Avenue 

New York, NY 10158-0012 

TEL: 212-692-6000 



International Joutnal of the 
Addictions (12/year) 
Marcel Dekker Journals 
270 Madison Avenue 
New York, NY 10016 
TEL: 212-696-9000 

JAMA: Journal of the American 
Medical Association (weekly) 
American Medical Association 
515 N. State St. 
Chicago, IL 60610 
TEL: 312-464-0183 

Journal of Addictive Diseases 
(quarterly) 

The Haworth Press, Inc. 
10 Alice Street 
Binghamton, NY 13904 
TEL: 800-3-HAWORTH 

* Journal of Adolescent Chemical 

Dependency (quarterly) 

The Haworth Press, Inc. 

10 Alice Street 

Binghamton, NY 13904 

TEL: 800-3^HAWORTH 

Journal of Adolescent Health Care 

(6/year) 

(Society for Adolescent Medicine) 
Elsevier Science Publishing Co., 
655 Avenue of the Americas 
New York, NY 10010 
TEL: r.. 2-989-5800 
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Journal of Adolescent Research 
(quarterly) 

Sage Publications, Inc. 
2455 Teller Road 
Newbury Park, CA 91320 
TEL: 805-499-0721 

* Journal of Alcohol and Drug 

Education (3/year) 

Alcohol & Drug Problems Assn of No. 

America 

1120 East Oakland 
P.O. Box 100212 
Lansing, ME 48901 
TEL: 517-484-0016 

Journal of American College Health 

(bimonthly) 

(American College Health Association) 
Heldref Publications 
4000 Albermarle St., NW 
Washington, D.C.. 20016 
TEL: 202-362-6445 

Journal of Applied Psychology 
(bimonthly) 

American Psychological Association 
1200 17th St., NW 
Washington, D.C. 20036 
TEL: 202-955-7600 

Journal of Child Psychology & 
Psychiatry & Applied Disciplines 

(7/year) 

(Assn of Child Psychology and 
Psychiatry) 

Journals Division Pergamon Press, Inc. 
Maxwell House, Fairview Park 
Elmsford,NY 10523 
TEL: 914-592-7700 



Journal of Consulting and Clinical 
Psychology (bimonthly) 
American Psychological Association 
1200 17th St.,NW 
Washington, D.C. 20036 
TEL: 202-955-7600 

Journal of Counseling & Development 

(bimonthly) 

American Assn for Counseling and 

Development 

5999 Stevenson Avenue 

Alexandria, VA 22304 

TEL: 703-823-9800 

* Journal of Drug Education 

(quarterly) 

Baywood Publishing Co., Inc. 

26 Austin Avenue 

P.O. Box 337 

Amity ville, NY 11701 

* Journal of Drug Issues (quarterly) 
Journal of Drug Issues, Inc. 

P.O. Box 4021 
Tallahassee, FL 32315-4021 

Journal of Public Health Policy 

(quarterly) 

Journal of Public Health Policy, Inc. 
208 Meadowood Drive 
South Burlington, VT 05403 
TEL: 802-658-0136 

* Journal of School Health (10/year) 
American School Health Association 
7263 State Route 43 

P.O. Box 708 
Kent, OH 44240 



Western Regional Center for Drug-Free Schools and Communities^ 
3.5 



R7 



Ev^uating AOD Program Implementation: A Consumer's Guide 



Journal of Studies on Alcohol 

(bimonthly) 

(Rutgers Center of Alcohol Studies) 
Alcohol Research Documentation, Inc. 
Box 969 

Piscataway, NJ 08855 
TEL: 908-932-2190 

Journa! of Substance Abuse (quarterly) 
Abiex Publishing Corporation 
355 Chestnut St. 
Norwood, NJ 07648 
TEL: 201-767-8450 



Preventive Medicine (bimonthly) 

(American Health Foundation) 

Academic Press, Inc. 

Journal Division 

1250 Sixth Avenue 

San Diego, CA 92101 

TEL: 619-230-1840 

Psychoiogsca! Reports (bimonthly) 
Dr. C.H. Ammons & Dr. R.B. Ammons, 
Eds. & Pubs. 
Box 9229 

Missoula, MT 59807 



Recent issues of these journals are available for use at the Western Regional Center for 
Drug- Free Schools and Communities at 101 SW Main, Portland, Oregon. 
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Journals on Program Evaluation 



Although the focus of these journals is evaluation rather than on evaluation of AOD 
programs, these journals provide discussions of current thinking and ideas in the field of 
evaluation. 



Educational Evaluation and Policy 
Analysis (quarterly) 
American Educational Research 
Association (AERA) 
1230 17th Street, NW 
Washington, D.C. 20036-3078 

Evaluation and Program Planning 

(quarterly) 

Pergamon Press, Inc. 
Journal Division 
Maxwell House 
Fairview Park 
Elmsford,NY 10523 
TEL: 914-592-7700 
FAX: 914-592-3625 



Evaluation Review (bimonthly) 
Sage Publications, Inc. 
2455 Teller Road 
Newbury Park, CA 91320 



New Directions for Program 
Evaluation (quarterly) 
(American Evaluation Association) 
Jossey-Bates Inc., Publishers 
350 Sansome St„ 5th Floor 
San Francisco, CA 94104 
TEL: 415-433-1767 
FAX: 415-433-0499 



Studies in Educational Evaluation 

(3/year) 

(UCLA Center for the Study of 
Evaluation) 
Pergamon Press, Inc. 
Journals Division 
Maxwell House 
Fairview Park 
Elmsford,NY 10523 
TEL: 914-592-7700 
FAX: 914-592-3625 



Evaluation Studies Review Annuals 
(annually) 

Sage Publications, Inc. 
2455 Teller Road 
Newbury Park, CA 91320 
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Information and Resource Centers 



A wealth of information on a variety of topics is available from the following national and 
regional centers. Check with these sources for updates on resources in evaluating AOD 
programs. 

!• OSAP's National CJearinghouse for Alcohol and Drug Information 
(ONCADI) 
P.O. Box 2345 
Rockville, MD 20847-2345 
1-800-729-6686 

Funded by the U.S. Department of Health and Human Services, ONCADI 

provides searches from its database on specific alcohol and other drug 
topics at no cost. It also disseminates a bimonthly publication titled: The 
OSAP Prevention Pipeline which contains prevention news, new program 
resources available from the Office for Substance Abuse Prevention and 
other federal agencies, as well as abstracts of recent journal articles and 
books in the alcohol and other drug field. The Western Regional Center 
for Drug-Free Schools aj)d Communities in Portland, Oregon, subscribes to 
The OSAP Prevention Pipeline. 

2. Educational Resources Information Center (ERIC) 

Dept. CCE 

1600 Research Boulevard 
Rockville, MD 20850 

Supported by the U.S. Department of Education, Office of Educational Research 
and Improvement (OERI), ERIC consists of 16 clearinghouses that 
respond to inquiries about its database, which is accessible to the public at 
more than 3,000 locations woridwide. Contact the Clearinghouse on 
Tests, Measurements, and Evaluation through the above toll-free number 
for citations on journal articles and documents related to educational 
evaluation. 



3, Western Regional Center for Drug-Free School and Communities 

Judith A. Johnson, Director 

A. Training and Technical Assistance on Evaluation 

Available at no cost to school districts in the West, contact the Western 
Regional Center office listed below that serves your state for further 
information. 
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Alaska, Idaho, Montana, Oregon, Washington, and Wyonriing: 
Carlos Sundermann/Rora Yen 
Northwest Regional Educational Laboratory 
101 SW Main Street, Suite 500 
PorUand,OR 97204 

503-275-9478 or 9497, 1-800-547-6339, ext. 478 or 497 

Hawaii and the Pacific Islands: 
Harvey Lee 
Honolulu Field Office 
1 164 Bishop Street, Suite 1409 
Honolulu, HI 96813 
808-532-1904 

Northern California and Northern Nevada: 
Ralph Baker 

Far West Laboratory for Educational Research and Development 

730 Harrison Street 

San Francisco, CA 94107-1242 

415-565-3000 

Southern California and Southern Nevada: 
Carol F. Thomas 
Southwest Regional Laboratory 
4665 Lampson Avenue 
Los Alamitos, California 90720 
310-598-7661 

Resource Center Services 

There are resource centers at each of the Western Regional Center offices 
(See 3A. above) that are accessible to administrators, teachers, curriculum 
specialists, alcohol and other drug coordinators, parents, and other 
preventionists to help them in their search for up-to-date information. 
Please contact the office in your area for further information and assistance 
The following types of services are offered: 

Library: A collection of AOD prevention curricula, programs, 
monographs, and videos is available free for a three week loan period. 
Telephone your requests to the Western Center or visit the Center. 

Western Center Publications: The Center disseminates to the public 
over 30 free publications developed by Center staff. These include the 
Western Center quanerly newsletter (mailed to over 16,000 clients). 
Prevention Research Updates (a publication that summarizes the recent 



Western Regional Center for Drug-free Schools and Communities^ 
3.9 



7i 



Evaluating ADD Program impiemer^tation: A Cor)sun)er's Guide 



research on AOD use and its prevention), and other publications of interest 
to preventionists on such topics as peer education, fetal effects of parental 
AOD use, summaries of successful prevention programs, and guides to 
student AOD surveys and parenting skills curricula. A listing of these 
publications is available upon request 

RADAR: The Center is a Regional Alcohol and Drug Awareness 
Resource (RADAR) Network member and thus has the current 
publications available through OSAFs National Clearinghouse for Alcohol 
and Drug Information (ONCADI). As a member of the network, the 
resource centers can di'aw upon the expertise of the other state, regional, 
and national organizations who are RADAR members. 

Organization Files: Files with catalogs, brochures, and program 
information are kept on organizations, publishers, agencies and individuals 
for reference use. 

Subject Files: Current newspaper, magazine, journal, and research articles 
are collected for reference use on a wide range of alcohol and other drug 
related topics. 

Newsletters, Journals: The Center subscribes to a variety of 
newsletters/journals. These can be used by the public at the Center. 

Western Regional Center Publications on Evaluation 

The following publications on evaluation, developed by Center staff, is 
available at no charge if you are in our service area. Western Regional 
Center services Alaska, American Samoa, California, Guam, Hawaii, 
Idaho, Montana, Nevada, Oregon, Palau, Washington, and Wyoming. 

Gabriel, R.M., & Brinkerhoff, C. (1990). Developing a communiry profile: 
A handbook for using pre-existing data in prevention planning. Portland, 
OR: Northwest Regional Educational Laboratory. 

Pollard, J.A., Gabriel, R.M., & Arter, J.A. (1991). Surveys of student 
alcohol and other drug use: A consumer's guide (2nd ed.) Portland, OR: 
Northwest Regional Educational Laboratory. 

Regional Drug Initiative. (1991). Drug impact index, 2nd edition. 
Portland, OR: Northwest Regional Educational Laboratory. 

Western Regional Center for Drug-Free Schools and Communities. (1991). 
Alcohol and other drug programs: Abstracts of selected evaluation 
studies. Portland, OR: Northwest Regional Educational Laboratory. 
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Western Regional Center for Drug-Free Schools and Connmunities. (1992), 
AOD program evaluation at the local level: A pilot site approach. Report 
No, 4, Portland, OR: Northwest Regional Educational Laboratory, 

Yen, F,B. & Brinkerhoff, C.V. (1992), Evaluation Resource Notebook for 
AOD Coordinators, Portland, OR: Northwest Regional Educational 
Laboratory, 
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